/var/spool/pbs/mom_priv/jobs/7504509.wlm01.SC: 10: /var/spool/pbs/mom_priv/jobs/7504509.wlm01.SC: module: not found
Sun Feb  6 05:02:00 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:06:00.0 Off |                    0 |
| N/A   42C    P0    48W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:07:00.0 Off |                    0 |
| N/A   46C    P0    70W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:0A:00.0 Off |                    0 |
| N/A   41C    P0    45W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:0B:00.0 Off |                    0 |
| N/A   39C    P0    43W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
/var/spool/pbs/mom_priv/jobs/7504509.wlm01.SC: 13: /var/spool/pbs/mom_priv/jobs/7504509.wlm01.SC: module: not found
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (313) bind mounts
stty: 'standard input': Inappropriate ioctl for device
Buffer just in case....


Executing quickruns...
switching root dir to  wsolevaluation-master
/scratch/users/ntu/ericotjo/wsolevaluation-master
not enough values to unpack (expected 2, got 0)
trying to use CUDA...
meta_path: metadata/ILSVRC/test
HeatmapComputerResnet50ILSVRC ILSVRC type 1...
ADJUSTING FOR OLDER PYTORCH VERSION!
0 val/ILSVRC2012_val_00000001.JPEG
1 val/ILSVRC2012_val_00000002.JPEG
2 val/ILSVRC2012_val_00000003.JPEG
3 val/ILSVRC2012_val_00000004.JPEG
4 val/ILSVRC2012_val_00000005.JPEG
5 val/ILSVRC2012_val_00000006.JPEG
6 val/ILSVRC2012_val_00000007.JPEG
7 val/ILSVRC2012_val_00000008.JPEG
8 val/ILSVRC2012_val_00000009.JPEG
9 val/ILSVRC2012_val_00000010.JPEG
16/50000                                                        
32/50000                                                        
48/50000                                                        
64/50000                                                        
80/50000                                                        
96/50000                                                        
112/50000                                                       
128/50000                                                       
144/50000                                                       
160/50000                                                       
176/50000                                                       
192/50000                                                       
208/50000                                                       
224/50000                                                       
240/50000                                                       
256/50000                                                       
272/50000                                                       
288/50000                                                       
304/50000                                                       
320/50000                                                       
336/50000                                                       
352/50000                                                       
368/50000                                                       
384/50000                                                       
400/50000                                                       
416/50000                                                       
432/50000                                                       
448/50000                                                       
464/50000                                                       
480/50000                                                       
496/50000                                                       
512/50000                                                       
Traceback (most recent call last):
  File "/mnt/xquickruns_resnet50.py", line 88, in <module>
    DEBUG_TOGGLES=[parse_bool_from_string(x) for x in args_dict['debug_toggles']])
  File "/mnt/xwsol/xresearch.py", line 118, in compute_resnet50_scoremaps
    layer_target=kwargs['scoremap_submode'])      
  File "/mnt/xwsol/xresearch.py", line 273, in save_resnet50_gradcam
    hmap = lgcam.attribute(x, y_pred) # torch.Size([1, 1, 12, 16])
  File "/opt/conda/lib/python3.6/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/captum/attr/_core/layer/grad_cam.py", line 197, in attribute
    attribute_to_layer_input=attribute_to_layer_input,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 599, in compute_layer_gradients_and_eval
    require_layer_grads=True,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 297, in _forward_layer_distributed_eval
    additional_forward_args=additional_forward_args,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/common.py", line 459, in _run_forward
    else inputs
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/model.py", line 29, in forward
    return self.backbone(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 202, in forward
    x = self.layer2(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/adjusted_model_component.py", line 83, in forward
    out = self.relu3(out)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 94, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 913, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 244.00 MiB (GPU 0; 15.75 GiB total capacity; 14.18 GiB already allocated; 79.88 MiB free; 477.82 MiB cached)
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (313) bind mounts
stty: 'standard input': Inappropriate ioctl for device
Buffer just in case....


Executing quickruns...
switching root dir to  wsolevaluation-master
/scratch/users/ntu/ericotjo/wsolevaluation-master
not enough values to unpack (expected 2, got 0)
trying to use CUDA...
meta_path: metadata/ILSVRC/test
HeatmapComputerResnet50ILSVRC ILSVRC type 1...
ADJUSTING FOR OLDER PYTORCH VERSION!
0 val/ILSVRC2012_val_00000001.JPEG
1 val/ILSVRC2012_val_00000002.JPEG
2 val/ILSVRC2012_val_00000003.JPEG
3 val/ILSVRC2012_val_00000004.JPEG
4 val/ILSVRC2012_val_00000005.JPEG
5 val/ILSVRC2012_val_00000006.JPEG
6 val/ILSVRC2012_val_00000007.JPEG
7 val/ILSVRC2012_val_00000008.JPEG
8 val/ILSVRC2012_val_00000009.JPEG
9 val/ILSVRC2012_val_00000010.JPEG
16/50000                                                        
32/50000                                                        
48/50000                                                        
64/50000                                                        
80/50000                                                        
96/50000                                                        
112/50000                                                       
128/50000                                                       
144/50000                                                       
160/50000                                                       
176/50000                                                       
192/50000                                                       
208/50000                                                       
224/50000                                                       
240/50000                                                       
256/50000                                                       
272/50000                                                       
288/50000                                                       
304/50000                                                       
320/50000                                                       
336/50000                                                       
352/50000                                                       
368/50000                                                       
384/50000                                                       
400/50000                                                       
416/50000                                                       
432/50000                                                       
448/50000                                                       
464/50000                                                       
480/50000                                                       
496/50000                                                       
512/50000                                                       
Traceback (most recent call last):
  File "/mnt/xquickruns_resnet50.py", line 88, in <module>
    DEBUG_TOGGLES=[parse_bool_from_string(x) for x in args_dict['debug_toggles']])
  File "/mnt/xwsol/xresearch.py", line 118, in compute_resnet50_scoremaps
    layer_target=kwargs['scoremap_submode'])      
  File "/mnt/xwsol/xresearch.py", line 277, in save_resnet50_gradcam
    hmap = lgcam.attribute(x, y_pred) # torch.Size([1, 1, 12, 16])
  File "/opt/conda/lib/python3.6/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/captum/attr/_core/layer/grad_cam.py", line 197, in attribute
    attribute_to_layer_input=attribute_to_layer_input,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 599, in compute_layer_gradients_and_eval
    require_layer_grads=True,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 297, in _forward_layer_distributed_eval
    additional_forward_args=additional_forward_args,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/common.py", line 459, in _run_forward
    else inputs
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/model.py", line 29, in forward
    return self.backbone(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 202, in forward
    x = self.layer2(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/adjusted_model_component.py", line 83, in forward
    out = self.relu3(out)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 94, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 913, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 244.00 MiB (GPU 0; 15.75 GiB total capacity; 14.18 GiB already allocated; 79.88 MiB free; 477.82 MiB cached)
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (313) bind mounts
stty: 'standard input': Inappropriate ioctl for device
Buffer just in case....


Executing quickruns...
switching root dir to  wsolevaluation-master
/scratch/users/ntu/ericotjo/wsolevaluation-master
not enough values to unpack (expected 2, got 0)
trying to use CUDA...
meta_path: metadata/ILSVRC/test
HeatmapComputerResnet50ILSVRC ILSVRC type 1...
ADJUSTING FOR OLDER PYTORCH VERSION!
0 val/ILSVRC2012_val_00000001.JPEG
1 val/ILSVRC2012_val_00000002.JPEG
2 val/ILSVRC2012_val_00000003.JPEG
3 val/ILSVRC2012_val_00000004.JPEG
4 val/ILSVRC2012_val_00000005.JPEG
5 val/ILSVRC2012_val_00000006.JPEG
6 val/ILSVRC2012_val_00000007.JPEG
7 val/ILSVRC2012_val_00000008.JPEG
8 val/ILSVRC2012_val_00000009.JPEG
9 val/ILSVRC2012_val_00000010.JPEG
16/50000                                                        
32/50000                                                        
48/50000                                                        
64/50000                                                        
80/50000                                                        
96/50000                                                        
112/50000                                                       
128/50000                                                       
144/50000                                                       
160/50000                                                       
176/50000                                                       
192/50000                                                       
208/50000                                                       
224/50000                                                       
240/50000                                                       
256/50000                                                       
272/50000                                                       
288/50000                                                       
304/50000                                                       
320/50000                                                       
336/50000                                                       
352/50000                                                       
368/50000                                                       
384/50000                                                       
400/50000                                                       
416/50000                                                       
432/50000                                                       
448/50000                                                       
464/50000                                                       
480/50000                                                       
496/50000                                                       
512/50000                                                       
Traceback (most recent call last):
  File "/mnt/xquickruns_resnet50.py", line 88, in <module>
    DEBUG_TOGGLES=[parse_bool_from_string(x) for x in args_dict['debug_toggles']])
  File "/mnt/xwsol/xresearch.py", line 118, in compute_resnet50_scoremaps
    layer_target=kwargs['scoremap_submode'])      
  File "/mnt/xwsol/xresearch.py", line 277, in save_resnet50_gradcam
    hmap = lgcam.attribute(x, y_pred) # torch.Size([1, 1, 12, 16])
  File "/opt/conda/lib/python3.6/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/captum/attr/_core/layer/grad_cam.py", line 197, in attribute
    attribute_to_layer_input=attribute_to_layer_input,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 599, in compute_layer_gradients_and_eval
    require_layer_grads=True,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 297, in _forward_layer_distributed_eval
    additional_forward_args=additional_forward_args,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/common.py", line 459, in _run_forward
    else inputs
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/model.py", line 29, in forward
    return self.backbone(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 203, in forward
    x = self.layer3(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/adjusted_model_component.py", line 69, in forward
    out = self.bn1(out)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1656, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 122.00 MiB (GPU 0; 15.75 GiB total capacity; 14.30 GiB already allocated; 53.88 MiB free; 381.51 MiB cached)
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (313) bind mounts
stty: 'standard input': Inappropriate ioctl for device
Buffer just in case....


Executing quickruns...
switching root dir to  wsolevaluation-master
/scratch/users/ntu/ericotjo/wsolevaluation-master
not enough values to unpack (expected 2, got 0)
trying to use CUDA...
meta_path: metadata/ILSVRC/test
HeatmapComputerResnet50ILSVRC ILSVRC type 1...
ADJUSTING FOR OLDER PYTORCH VERSION!
0 val/ILSVRC2012_val_00000001.JPEG
1 val/ILSVRC2012_val_00000002.JPEG
2 val/ILSVRC2012_val_00000003.JPEG
3 val/ILSVRC2012_val_00000004.JPEG
4 val/ILSVRC2012_val_00000005.JPEG
5 val/ILSVRC2012_val_00000006.JPEG
6 val/ILSVRC2012_val_00000007.JPEG
7 val/ILSVRC2012_val_00000008.JPEG
8 val/ILSVRC2012_val_00000009.JPEG
9 val/ILSVRC2012_val_00000010.JPEG
16/50000                                                        
32/50000                                                        
48/50000                                                        
64/50000                                                        
80/50000                                                        
96/50000                                                        
112/50000                                                       
128/50000                                                       
144/50000                                                       
160/50000                                                       
176/50000                                                       
192/50000                                                       
208/50000                                                       
224/50000                                                       
240/50000                                                       
256/50000                                                       
272/50000                                                       
288/50000                                                       
304/50000                                                       
320/50000                                                       
336/50000                                                       
352/50000                                                       
368/50000                                                       
384/50000                                                       
400/50000                                                       
416/50000                                                       
432/50000                                                       
448/50000                                                       
464/50000                                                       
480/50000                                                       
496/50000                                                       
512/50000                                                       
Traceback (most recent call last):
  File "/mnt/xquickruns_resnet50.py", line 88, in <module>
    DEBUG_TOGGLES=[parse_bool_from_string(x) for x in args_dict['debug_toggles']])
  File "/mnt/xwsol/xresearch.py", line 118, in compute_resnet50_scoremaps
    layer_target=kwargs['scoremap_submode'])      
  File "/mnt/xwsol/xresearch.py", line 277, in save_resnet50_gradcam
    hmap = lgcam.attribute(x, y_pred) # torch.Size([1, 1, 12, 16])
  File "/opt/conda/lib/python3.6/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/captum/attr/_core/layer/grad_cam.py", line 197, in attribute
    attribute_to_layer_input=attribute_to_layer_input,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 599, in compute_layer_gradients_and_eval
    require_layer_grads=True,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 297, in _forward_layer_distributed_eval
    additional_forward_args=additional_forward_args,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/common.py", line 459, in _run_forward
    else inputs
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/model.py", line 29, in forward
    return self.backbone(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 203, in forward
    x = self.layer3(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/adjusted_model_component.py", line 74, in forward
    out = self.relu2(out)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/activation.py", line 94, in forward
    return F.relu(input, inplace=self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 913, in relu
    result = torch.relu(input)
RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 15.75 GiB total capacity; 14.36 GiB already allocated; 21.88 MiB free; 352.54 MiB cached)
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (313) bind mounts
stty: 'standard input': Inappropriate ioctl for device
Buffer just in case....


Executing quickruns...
switching root dir to  wsolevaluation-master
/scratch/users/ntu/ericotjo/wsolevaluation-master
not enough values to unpack (expected 2, got 0)
trying to use CUDA...
meta_path: metadata/ILSVRC/test
HeatmapComputerResnet50ILSVRC ILSVRC type 1...
ADJUSTING FOR OLDER PYTORCH VERSION!
0 val/ILSVRC2012_val_00000001.JPEG
1 val/ILSVRC2012_val_00000002.JPEG
2 val/ILSVRC2012_val_00000003.JPEG
3 val/ILSVRC2012_val_00000004.JPEG
4 val/ILSVRC2012_val_00000005.JPEG
5 val/ILSVRC2012_val_00000006.JPEG
6 val/ILSVRC2012_val_00000007.JPEG
7 val/ILSVRC2012_val_00000008.JPEG
8 val/ILSVRC2012_val_00000009.JPEG
9 val/ILSVRC2012_val_00000010.JPEG
16/50000                                                        
32/50000                                                        
48/50000                                                        
64/50000                                                        
80/50000                                                        
96/50000                                                        
112/50000                                                       
128/50000                                                       
144/50000                                                       
160/50000                                                       
176/50000                                                       
192/50000                                                       
208/50000                                                       
224/50000                                                       
240/50000                                                       
256/50000                                                       
272/50000                                                       
288/50000                                                       
304/50000                                                       
320/50000                                                       
336/50000                                                       
352/50000                                                       
368/50000                                                       
384/50000                                                       
400/50000                                                       
416/50000                                                       
432/50000                                                       
448/50000                                                       
464/50000                                                       
480/50000                                                       
496/50000                                                       
512/50000                                                       
Traceback (most recent call last):
  File "/mnt/xquickruns_resnet50.py", line 88, in <module>
    DEBUG_TOGGLES=[parse_bool_from_string(x) for x in args_dict['debug_toggles']])
  File "/mnt/xwsol/xresearch.py", line 118, in compute_resnet50_scoremaps
    layer_target=kwargs['scoremap_submode'])      
  File "/mnt/xwsol/xresearch.py", line 277, in save_resnet50_gradcam
    hmap = lgcam.attribute(x, y_pred) # torch.Size([1, 1, 12, 16])
  File "/opt/conda/lib/python3.6/site-packages/captum/log/__init__.py", line 35, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/captum/attr/_core/layer/grad_cam.py", line 197, in attribute
    attribute_to_layer_input=attribute_to_layer_input,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 599, in compute_layer_gradients_and_eval
    require_layer_grads=True,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/gradient.py", line 297, in _forward_layer_distributed_eval
    additional_forward_args=additional_forward_args,
  File "/opt/conda/lib/python3.6/site-packages/captum/_utils/common.py", line 459, in _run_forward
    else inputs
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/model.py", line 29, in forward
    return self.backbone(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 203, in forward
    x = self.layer3(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/xwsol/adjusted_model_component.py", line 73, in forward
    out = self.bn2(out)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 81, in forward
    exponential_average_factor, self.eps)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py", line 1656, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 15.75 GiB total capacity; 14.33 GiB already allocated; 31.88 MiB free; 375.03 MiB cached)
======================================================================================

			Resource Usage on 2022-02-06 05:12:21.418413:

	JobId: 7504509.wlm01  
	Project: 12001577 
	Exit Status: 1
	NCPUs Requested: 20				NCPUs Used: 20
							CPU Time Used: 02:35:21
	Memory Requested: 64gb 				Memory Used: 3017320kb
							Vmem Used: 35726224kb
	Walltime requested: 24:00:00 			Walltime Used: 00:10:21
	
	Execution Nodes Used: (dgx4101:ncpus=20:ngpus=4:mem=67108864kb)
	
 ======================================================================================
